The cell below produces a button that toggles code in this notebook on and off.
from IPython.display import HTML
HTML('''<script>
code_show=true;
function code_toggle() {
if (code_show){
$('div.input').hide();
} else {
$('div.input').show();
}
code_show = !code_show
}
$( document ).ready(code_toggle);
</script>
<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code."></form>''')
$$ \text{model}\left(x,\Theta\right) = w_0 + xw_1 $$
# simple linear model using numpy 'dot' for quick multiplication
# params packaged as theta = [w_0,w_1,...]
def model(x,theta):
# compute linear combination of input
a = theta[0] + np.dot(x.T,theta[1:])
return a.T
$\,$ $\,$ $\,$
a typical origin story for RNNs echoed throughout the internet goes like this
$$ \text{model}\left(x,\Theta\right) = w_0 + xw_1 $$
$$ \text{model}\left(x,\Theta\right) = w_0 + xw_1 + x^2w_2 + \cdots + w_Bx^B $$
$$ \text{model}\left(x,\Theta\right) = w_0 + \text{tanh}\left(w_{1,0} + xw_{1,1}\right)w_1 + \cdots + \text{tanh}\left(w_{1,B} + xw_{1,B}\right)w_B $$
# model employing fully connected network
# params packaged as: theta = [internal_network, final_linear_combo]
def model(x,theta):
# pass input through network
f = network_pass(x,theta[0])
# compute final linear combination and return
a = theta[1][0] + np.dot(f.T,theta[1][1:])
return a.T
# guts of a fully connected network
def network_pass(a, w):
# loop through each layer matrix
for W in w:
# compute inner product with current layer weights
a = W[0] + np.dot(a.T, W[1:])
# output of layer activation
a = activation(a).T
return a
import sys
sys.path.append('../')
import numpy as np
import library.animators as anim
# This is needed to compensate for %matplotlib notebook's tendancy to blow up images when plotted inline
%matplotlib notebook
from matplotlib import rcParams
rcParams['figure.autolayout'] = True
%load_ext autoreload
%autoreload 2
$$ x_1,\,x_2,\,x_3,\,...,x_{t},\,... $$
$$ h_t = f\left(x_t\right) $$
\begin{array} \ \text{sum of the first $1$ elements:} \,\,\,\,\,\,\,\,\,\,\,\,\, h_1 = x_1 \\ \text{sum of the first $2$ elements:} \,\,\,\,\,\,\,\,\,\,\,\,\, h_2 = x_1 + x_2 \\ \text{sum of the first $3$ elements:} \,\,\,\,\,\,\,\,\,\,\,\,\, h_3 = x_1 + x_2 + x_3 \\ \text{sum of the first $4$ elements:} \,\,\,\,\,\,\,\,\,\,\,\,\, h_4 = x_1 + x_2 + x_3 + x_4 \\ \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \vdots \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \vdots \\ \text{sum of the first $t$ elements:} \,\,\,\,\,\,\,\,\,\,\,\,\, h_{t} = x_1 + x_2 + x_3 + x_4 + \cdots + x_t \\ \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \vdots \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \vdots \\ \end{array}
obviously super wasteful
the right way: a simple recursion
\begin{array} \ \text{sum of the first $1$ elements:} \,\,\,\,\,\,\,\,\,\,\,\,\, h_1 = x_1 \\ \text{sum of the first $2$ elements:} \,\,\,\,\,\,\,\,\,\,\,\,\, h_2 = x_1 + x_2 \\ \text{sum of the first $3$ elements:} \,\,\,\,\,\,\,\,\,\,\,\,\, h_3 = x_1 + x_2 + x_3 \\ \text{sum of the first $4$ elements:} \,\,\,\,\,\,\,\,\,\,\,\,\, h_4 = x_1 + x_2 + x_3 + x_4 \\ \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \vdots \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \vdots \\ \text{sum of the first $t$ elements:} \,\,\,\,\,\,\,\,\,\,\,\,\, h_{t} = x_1 + x_2 + x_3 + x_4 + \cdots + x_t \\ \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \vdots \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \vdots \\ \end{array}
\begin{array} \ \text{sum of the first $1$ elements:} \,\,\,\,\,\,\,\,\,\,\,\,\, h_1 = x_1 \\ \text{sum of the first $2$ elements:} \,\,\,\,\,\,\,\,\,\,\,\,\, h_2 = h_1 + x_2 \\ \text{sum of the first $3$ elements:} \,\,\,\,\,\,\,\,\,\,\,\,\, h_3 = h_2 + x_3 \\ \text{sum of the first $4$ elements:} \,\,\,\,\,\,\,\,\,\,\,\,\, h_4 = h_3 + x_4 \\ \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \vdots \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \vdots \\ \text{sum of the first $t$ elements:} \,\,\,\,\,\,\,\,\,\,\,\,\, h_{t} = h_{t-1} + x_t \\ \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \vdots \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \vdots \\ \end{array}
$$ h_t = h_{t-1} + x_t = x_1 + x_2 + \cdots + x_t $$
$$ h_t = f\left(h_{t-1},x_t \right) $$
$$ h_t = f\left(x_t\right) $$
$$ h_t = f\left(h_{t-1},x_t \right) $$
$$ h_t = \alpha h_{t-1} + (1 - \alpha) x_t $$
$\,\,\,$ where $0 \leq \alpha \leq 1$